Skip to content

Conversation

dejan2609
Copy link
Contributor

@dejan2609 dejan2609 commented Apr 18, 2025

Related JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-19174

List of changes:

  • prerequisite Jira ticket:
  • mandatory version upgrades:
    • Gradle version: 8.14.3 -->> 9.0.0 9.1.0
    • Gradle Shadow plugin: 8.3.6 -->> 9.0.2 8.3.9
  • build logic will be refactored to accommodate Gradle 9 breaking
    changes
  • (optional): a dozen of Gradle plugins versions will also be upgraded
  • other JIRA tickets that had to be solved all along:

Related links:

Notes:

@github-actions github-actions bot added triage PRs from the community build Gradle build or GitHub Actions small Small PRs labels Apr 18, 2025
@dejan2609 dejan2609 changed the title [KAFKA-19174] Gradle version upgrade 8 -->> 9 [Work in progress] [KAFKA-19174] Gradle version upgrade 8 -->> 9 Apr 18, 2025
Copy link

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Copy link

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions bot added stale Stale PRs and removed needs-attention stale Stale PRs labels Jul 25, 2025
Copy link

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@dejan2609
Copy link
Contributor Author

Update: project compilation works as expected (when powered by Gradle 9) but binary release creation fails.
More work has to be done in order to accommodate to Gradle breaking changes.

Copy link

github-actions bot commented Aug 3, 2025

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@chia7712
Copy link
Member

chia7712 commented Aug 6, 2025

@dejan2609 any update?

Copy link

github-actions bot commented Aug 8, 2025

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

@dejan2609
Copy link
Contributor Author

@chia7712
Will provide an update during the weekend (there are two problems with Gradle 9 build, will explain in more details and provide some insights).

@chia7712
Copy link
Member

chia7712 commented Aug 8, 2025

@dejan2609 thanks!

@dejan2609
Copy link
Contributor Author

@chia7712 feel free to check this related PR: #20327

@dejan2609
Copy link
Contributor Author

Pending: I will also explain second problem (specific to Gradle 9, that is).

Copy link

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

…it should be clearly visible now which submodules configurations had to be extended)
@dejan2609
Copy link
Contributor Author

Update: an additional commit has been pushed (to polish some things).

I also did some testing (locally, on my laptop):

  1. ./gradlew :jmh-benchmarks:shadowJar
  2. Create a distribution and test it:
    • ./gradlew releaseTarGz
    • unpack (with tar xvfz) /kafka/core/build/distributions/kafka_2.13-4.2.0-SNAPSHOT.tgz
    • test number of files: find kafka_2.13-4.2.0-SNAPSHOT -type f | wc -l -->> yields the same number of files as for the trunk distribution ✅
    • test like here: https://kafka.apache.org/quickstart -->> ✅

Also, I revisited some previous shadow plugin related testing:

Testing procedure:

  • git checkout trunk
  • change gradle.properties version to 4.2.0.1-SNAPSHOT
  • execute ./gradlew :clients:publishToMavenLocal -PskipSigning=true
  • git reset --hard
  • git checkout KAFKA-19174
  • change gradle.properties version to 4.2.0.2-SNAPSHOT
  • execute (again) ./gradlew :clients:publishToMavenLocal -PskipSigning=true

Test results:

  • compare two .pom file versions:
diff ~/.m2/repository/org/apache/kafka/kafka-clients/4.2.0.1-SNAPSHOT/kafka-clients-4.2.0.1-SNAPSHOT.pom ~/.m2/repository/org/apache/kafka/kafka-clients/4.2.0.2-SNAPSHOT/kafka-clients-4.2.0.2-SNAPSHOT.pom
6c6
<   <version>4.2.0.1-SNAPSHOT</version>
---
>   <version>4.2.0.2-SNAPSHOT</version>
  • unzip both jar files: ~/.m2/repository/org/apache/kafka/kafka-clients/4.2.0.X-SNAPSHOT/kafka-clients-4.2.0.X-SNAPSHOT.jar: both META-INF/MANIFEST.MF files contain only one line:
Manifest-Version: 1.0
dependencies {
    //implementation("org.apache.kafka:kafka-clients:4.2.0.1-SNAPSHOT")
    implementation("org.apache.kafka:kafka-clients:4.2.0.2-SNAPSHOT")
}
  • create two files and compare them
    • ./gradlew dependencies > deps-trunk.txt
    • ./gradlew dependencies > deps-PR.txt
    • files are equivalent (i.e. Gradle dependencies trees match in everything except for the kafka-clients version(s))

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 21, 2025

⚠️ Signing is another thing that could be tested (in addition to those performed above).

@chia7712 let me know what you think: is this type of test required ?

@chia7712
Copy link
Member

@dejan2609 could you please take a look at my previous comment? #19513 (comment)

@dejan2609
Copy link
Contributor Author

@dejan2609 could you please take a look at my previous comment? #19513 (comment)

Sure, I am creating POC to explain this case.

APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excuse me, why to have those changes? Are those changed generated by gradle 9?

@chia7712
Copy link
Member

@dejan2609 It would be helpful to understand the reasoning behind the changes in this PR, since introducing unnecessary modifications is usually not a good idea. That’s why I left some questions on parts where I couldn’t see the motivation—for example: #19513 (comment)

Upgrading Gradle is an important task for Kafka, so I’d appreciate your help in clarifying the intent of this PR. Thanks!

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 25, 2025

@dejan2609 It would be helpful to understand the reasoning behind the changes in this PR, since introducing unnecessary modifications is usually not a good idea. That’s why I left some questions on parts where I couldn’t see the motivation—for example: #19513 (comment)

Upgrading Gradle is an important task for Kafka, so I’d appreciate your help in clarifying the intent of this PR. Thanks!

Yes, I understand the gravity of the situation. It goes without saying that these types of upgrades must be done with high caution. I have prior experience with upgrading Gradle major versions in Kafka (from Gradle 6 to 7 and then from 7 to 8) and they were complicated enough (but both of those upgrades pale in comparison to this one).

I left answers to both comments above, but I will sum it up here also:

  • first commit 79f0415 resolves bug KAFKA-19591 (on Gradle 8 level); objectively: this commit is a prerequisite for a Gradle 8 to 9 upgrade
  • all other commits are exclusively related to Gradle 8 to 9 upgrade

I opted to keep separate commits for every phase just for the sake of clarity (and that was a good decision, I guess).

Options to proceed:

  • option [1]:
    • merge commit 79f0415 (KAFKA-19591) into trunk separately
    • squash all other commits (related to Gradle 8 -->> 9 upgrade) into one and, after few days, merge that commit into trunk
  • option [2]: squash all commits into one
  • option [3]:
    • put aside commit 79f0415 (KAFKA-19591) and try to concentrate only on Gradle 8 -->> 9 upgrade

Note: not sure about the possibility of using option [3]: from where I see it 79f0415 (KAFKA-19591) is a must (either separately or as a part of upgrade); in more details: this is a minor bug on Gradle 8 level but could be a blocker for upgrade (if not included) on Gradle 9 level.

Let me know what you think @chia7712 (I edited this comment few times, hence this note).

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 25, 2025

@chia7712 Come to think of it: there is an option [4] also: I'll create a POC to narrow down options mentioned above 🙂

Stay tuned, I'll send an update here ASAP.

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 26, 2025

Ok, I'll split this POC into multiple comments, starting from my local environment:

dejan@dejan-HP-ProBook-450-G7:~/kafka$ java -version 
openjdk version "17.0.16" 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ javac -version 
javac 17.0.16
dejan@dejan-HP-ProBook-450-G7:~/kafka$ sdk list gradle     
================================================================================
Available Gradle Versions
================================================================================
   * 9.1.0               7.4                 5.6.1               3.2.1          
   * 9.0.0               7.3.3               5.6                 3.2            
 > * 8.14.3              7.3.2               5.5.1               3.1            
     8.14.2              7.3.1               5.5                 3.0            
   * 8.14.1              7.3                 5.4.1               2.14.1         
     8.14                7.2                 5.4                 2.14           
     8.13                7.1.1               5.3.1               2.13           
     8.12.1              7.1                 5.3                 2.12           
     8.12                7.0.2               5.2.1               2.11           
     8.11.1              7.0.1               5.2                 2.10           
     8.11                7.0                 5.1.1               2.9            
     8.10.2              6.9.4               5.1                 2.8            
     8.10.1              6.9.3               5.0                 2.7            
     8.10                6.9.2               4.10.3              2.6            
     8.9                 6.9.1               4.10.2              2.5            
   * 8.8                 6.9                 4.10.1              2.4            
   * 8.7                 6.8.3               4.10                2.3            
     8.6                 6.8.2               4.9                 2.2.1 

Overview of gradlew changes in Kafka trunk:
image


Related GitHub Gradle links for unixStartScript.txt (file that is used as a template for gradlew generation):

(the thing is that Gradle decided to change the path for unixStartScript.txt (template for gradlew) in Gradle version 8.8 (and hence now Kafka's trunk gradlew is referencing version 8.7 template)

🪧Loosely related Gradle issue: gradle/gradle#30101

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 26, 2025

Now that we have an environment in place, we need to do the following things:

  • checkout Kafka trunk
  • make sure that there are no changes whatsoever
  • remove all files that are not committed/pushed into GitHub repo (we want to make sure that we don't have gradle/wrapper/gradle-wrapper.jar - hence we are forced to use our local Gradle installation)
  • use our local Gradle installation (same version as in trunk: 8.14.3)
  • execute: gradle wrapper -q
  • review git diff 🙂 ➡️ KAFKA-19591 POC: fix gradlew (add missing changes) #20599
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git diff
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git status
On branch trunk
Your branch is up-to-date with 'origin/trunk'.

nothing to commit, working tree clean
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git clean -fdx
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ls gradle/wrapper/
gradle-wrapper.properties
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git log -5 --oneline 
9e9d2a23ef (HEAD -> trunk, upstream/trunk, upstream/HEAD, origin/trunk, origin/HEAD) MINOR: fix flaky sys test for static membership (#20594)
857b1e92cc KAFKA-19719: --no-initial-controllers should not assume kraft.version=1 (#20551)
f4e00e9cf0 MINOR: Remove unnecessary check in ReplicaManager (#20588)
97c8c6b595 KAFKA-19733 Fix arguments to assertEquals() in clients module (#20586)
14917ae727 MINOR: Handle envelope response in AutoTopicCreationManager (#20569)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ gradle -version

------------------------------------------------------------
Gradle 8.14.3
------------------------------------------------------------

Build time:    2025-07-04 13:15:44 UTC
Revision:      e5ee1df3d88b8ca3a8074787a94f373e3090e1db

Kotlin:        2.0.21
Groovy:        3.0.24
Ant:           Apache Ant(TM) version 1.10.15 compiled on August 25 2024
Launcher JVM:  17.0.16 (Ubuntu 17.0.16+8-Ubuntu-0ubuntu124.04.1)
Daemon JVM:    /usr/lib/jvm/java-17-openjdk-amd64 (no JDK specified, using current Java home)
OS:            Linux 6.8.0-84-generic amd64

dejan@dejan-HP-ProBook-450-G7:~/kafka$ gradle wrapper -q
Starting build with version 4.2.0-SNAPSHOT (commit id 9e9d2a23) using Gradle 8.14.3, Java 17 and Scala 2.13.16
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git status
On branch trunk
Your branch is up-to-date with 'origin/trunk'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   gradlew

no changes added to commit (use "git add" and/or "git commit -a")
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git diff
diff --git a/gradlew b/gradlew
index f4bb3360e1..0eaf5d7382 100755
--- a/gradlew
+++ b/gradlew
@@ -15,6 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# SPDX-License-Identifier: Apache-2.0
+#
 
 ##############################################################################
 #
@@ -55,7 +57,7 @@
 #       Darwin, MinGW, and NonStop.
 #
 #   (3) This script is generated from the Groovy template
-#       https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
+#       https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
 #       within the Gradle project.
 #
 #       You can find Gradle at https://github.com/gradle/gradle/.
@@ -84,7 +86,7 @@ done
 # shellcheck disable=SC2034
 APP_BASE_NAME=${0##*/}
 # Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
-APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD=maximum
@@ -112,20 +114,7 @@ case "$( uname )" in                #(
   NONSTOP* )        nonstop=true ;;
 esac
 
-
-# Loop in case we encounter an error.
-for attempt in 1 2 3; do
-  if [ ! -e "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" ]; then
-    if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v8.14.3/gradle/wrapper/gradle-wrapper.jar"; then
-      rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
-      # Pause for a bit before looping in case the server throttled us.
-      sleep 5
-      continue
-    fi
-  fi
-done
-
-CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
+CLASSPATH="\\\"\\\""
 
 
 # Determine the Java command to use to start the JVM.
@@ -216,7 +205,7 @@ fi
 DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
 
 # Collect all arguments for the java command:
-#   * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
+#   * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
 #     and any embedded shellness will be escaped.
 #   * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
 #     treated as '${Hostname}' itself on the command line.
@@ -224,7 +213,20 @@ DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
 set -- \
         "-Dorg.gradle.appname=$APP_BASE_NAME" \
         -classpath "$CLASSPATH" \
-        org.gradle.wrapper.GradleWrapperMain \
+
+# Loop in case we encounter an error.
+for attempt in 1 2 3; do
+  if [ ! -e "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" ]; then
+    if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v8.14.3/gradle/wrapper/gradle-wrapper.jar"; then
+      rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
+      # Pause for a bit before looping in case the server throttled us.
+      sleep 5
+      continue
+    fi
+  fi
+done
+
+        -jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \
         "$@"
 
 # Stop when "xargs" is not available.
(END)

@dejan2609
Copy link
Contributor Author

dejan2609 commented Sep 26, 2025

@chia7712 you can review this again.

To sum it up (and please pardon me for such a long comments above, these things are quite tricky to catch and hence I wanted to provide you as much details as possible):

  • root cause for this issue is Gradle 8.8 path change for the template file unixStartScript.txt (that is used by Gradle to generate gradlew)
  • how this issue affects Gradle wrapper: Gradle build still works, but it is only a question of time when this escalates due to a fact that gradlew is missing all recent changes brought by a template file unixStartScript.txt (i.e. at the moment Gradle trunk Gradle wrapper version 8.14.3 is referencing 8.7 template file content)
  • the issue is tricky to find: if we are using gradle wrapper only (i.e. gradle wrapper jar older versions) then this issue stays hidden
  • local Gradle installation was a way to go for me to find and fix this

Where to proceed from here:

  1. merge this commit for KAFKA-19591 separately or
  2. bundle it with Gradle 8 to 9 upgrade or
  3. something else - not sure about other options, but I'm open for ideas/suggestions

Note: maybe it would be a good idea to describe this corner case (for Gradle wrapper generation, that is) in README.md file.

Signing off, will get back in 3 or 4 days 🥾 ⛰️ 👋

@dejan2609
Copy link
Contributor Author

dejan2609 commented Oct 1, 2025

Hi @chia7712 ! I did some cleanup (resolved old/outdated conversations).

You probably have a lot of things in your schedule, but please try to squeeze this in somehow.

TLDR version for you:

  • a distribution Gradle submodule is required (but that solution is as light as possible: paths for generated tgz files stay the same)
  • as for gradlew changes (KAFKA-19591): they are in fact a fix (missing pieces) generated on Gradle 8 level - see my detailed explanation above (and from my POV it should be solved - either separately or as a part of this Gradle 8 to 9 upgrade)

Waiting for your input on this.

@chia7712
Copy link
Member

chia7712 commented Oct 1, 2025

@dejan2609 I open the https://issues.apache.org/jira/browse/KAFKA-19750 to describe the root cause of releaseTarGz. I think the simple way to fix it is to remove project(':tools:tools-api') from releaseTarGz, since it is already in runtime path of core module. Also, it would be helpful to leave comment on the implementation project(':tools:tools-api') to remind developers to add project(':tools:tools-api') back if project(':tools:tools-api') is removed from the runtime path

@dejan2609
Copy link
Contributor Author

@dejan2609 I open the https://issues.apache.org/jira/browse/KAFKA-19750 to describe the root cause of releaseTarGz. I think the simple way to fix it is to remove project(':tools:tools-api') from releaseTarGz, since it is already in runtime path of core module. Also, it would be helpful to leave comment on the implementation project(':tools:tools-api') to remind developers to add project(':tools:tools-api') back if project(':tools:tools-api') is removed from the runtime path

Got it. Should I add a commit that resolves that issue here or you want to solve it separately ?

@chia7712
Copy link
Member

chia7712 commented Oct 1, 2025

Got it. Should I add a commit that resolves that issue here or you want to solve it separately ?

What do you think of this approache? Is it a viable replacement for the "distribution" module? If yes, it will be great this PR include the approach. If not, we could have a separate PR for that to fix the warnings for other active branches.

@dejan2609
Copy link
Contributor Author

Aha, now I see it clearly 💡
It goes without saying that we should avoid a :distribution module (it that's possible).

Allow me some time to test this idea, will update you asap.

@dejan2609
Copy link
Contributor Author

I may have some good news for you @chia7712 !
Closing the gap on this issue: we will likely be able to drop Gradle helper :submodule 🤞

So, I just scrapped a skeleton POC solution #20625 with just two things:

  • Gradle 8 -->> 9 upgrade
  • project(':tools:tools-api').configurations.runtimeClasspath removed out of releaseTarGz task

Results look encouraging: releaseTarGz task works fine (same goes for :jmh-benchmarks:shadowJar) ✅

Action points for me would be to reorganize all remaining commits into a new solution (that keeps releaseTarGz task in :core) and perform some tests (should be doable in a few days.).

If I may suggest something (my 2 cents on this): we could opt to solve KAFKA-19750 separately (i.e. on a Gradle 8 level).
Rationale: it would be much easier to spot that single isolated change (and revert it back if/when the time comes) rather than bundle it here.
But it's your call, let me know which option you prefer.

All-in-all: it is safe to say that things are wrapping up.
Stay tuned, we are so close 🎯 🙂

@chia7712
Copy link
Member

chia7712 commented Oct 2, 2025

If I may suggest something (my 2 cents on this): we could opt to solve KAFKA-19750 separately (i.e. on a Gradle 8 level).
Rationale: it would be much easier to spot that single isolated change (and revert it back if/when the time comes) rather than bundle it here.

You are right. @brandboat Do you have free time to file a patch for it first?

@brandboat
Copy link
Member

Many thanks to the help @chia7712, @dejan2609. Here is the pr link: #20628

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Gradle build or GitHub Actions ci-approved tools
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants